Flowpaths by Response Time

This feature is used to view the Flowpaths by response time. To do this, follow the below- mentioned steps:

Right-click on a tier, go to DrillDown, and click Flowpaths by Response Time option as shown in Figure 45.

The Flowpath report is displayed as shown in Figure 46. Flowpaths are sorted based on the response time in descending order, meaning the flow path with the highest response time is displayed at the top and so on.

Flowpath contains all the details of business transactions requests made by real time users at a specific time interval and the path taken by application traffic between two points anywhere on the network. User can sort the flowpath on the bases of fields in ascending or descending order as per requirement by clicking on field name. Following columns are available in flowpath response by time:

Tier: This denotes the name of the tier.
Server: Denotes the name of the server.
Instance: This denotes the instance running on that tier.
CallOutErrors: It denotes the number of errors that occurred in the method invocation. On clicking this link, user can view exceptions for error callout.
Business Transaction: Shows the name of the business transaction executed on the instance, such as catalog, product, or tracking an order. By clicking this, user can view the IP summary report.
URL: Shows the page URL of the business transaction. User can view HTTP reports by using this feature.
Category: Denotes the category of business transaction, such as normal, slow, very slow.
Start Time: Denotes the start time of the business transaction or method.
Total Response Time: The time taken in the execution of a business transaction. It is in MM/DD/YY HH:MM: SS format. By clicking this, user can view the method timing reports.
CPU Time(m/s): Time spent by CPU to execute transactions.
Methods: It shows the number of methods executed for that business transaction. By clicking on this user can view method details reports.
Callouts: Shows the number of calls for that business transaction.
DB callout: It shows the database call counts.
Status: It shows the status of the response.
Flag: It shows the feature related information when it is being enabled. Currently, the user can see the feature of the following: DL, HTTP Body Capturing, and Havoc.
WaitTime(ms): Any thread consuming time on wait method.
SyncTime(ms): Any thread stuck before synchronized block or synchronized method due to lock not being available.
CorrID: This denotes the correlation ID.

Apart from this, the flowpath window displays various icons through which a user can perform various functions:

SR. No.	Icon	Description
1		Copy flowpath link
2		Show all flowpaths of selected flowpath’s session
3		Show all flowpaths of selected flowpath’s NV session
4		Open NV Session
5		Open Executive Dashboard
6		Open aggregate flowpaths of selected flowpaths
7		Download as PDF report
8		Download as Word report
9		Download as Excel report
10		Start auto instrumentation
11		Compare flowpaths
12		Delete Cached Data
13		Logs

Note: A user can perform the following operations in the flowpath window which are listed below:

Toggle Column: On selecting this, the user can show/hide a particular column.
Column Filter: On selecting this, the user can filter out the columns as per the following: Business Transactions, URL, Category, Start Time, Total Response Time, CPU Time, Methods, Callouts, DB Callouts, Status, and Flag.
Search: On selecting this, the user can also search a flowpath report.
The above-described icons are same for all the flowpaths.

Split View Flowpath Report

You can view the records in split view format (by clicking the hyperlinks, such as Business Transactions, URL, and Category in the Flowpath report) where flowpath details are listed in a single place. Split View Report displays all other reports in DDR in a single place and you can see all the reports related to the same flowpath instance in a single flow. This report helps in observing all the details related to the same flowpath instance. Through this report, you can further drill down to other reports.

First Row: BT name, category, start time, response time.
Second Row: Tier Name, Server Name, Instance Name.
Third Row: Icons with values (only non-zero values and icons) and status code.

Integrated flowpath Report

An Integrated Flowpath Report is useful in case you want to see all reports in one place. You can see all the reports related to a single flowpath instance in one place.

Now, there is no need to traverse through breadcrumb to see all the reports related to a single flowpath instance. This saves time for you and helps in observing the data in a better way.

There are the following integrated flowpath reports:

Method Call Details
HotSpot
DB Queries
HTTP Report
Exception
Method Timing
IP Summary
Transaction Flowmap
Sequence Diagram

Note: The details of all these reports are provided in later sections.

Method Call Details

The user can access method and thread level execution details for the transaction in the form of method calling tree. This section displays a table of all the methods that are called in the business transactions. It displays method details, such as elapsed time, wait time, percentage, CPU time, and so on.

It displays method calling tree that displays details of all the executed methods. In this we have the following columns:

Method: It denotes the number of methods executed for that business transaction.
Flag: It is a representation of where DL(dynamic logging) is applied.
Elapsed Time: It is the amount of time that passes from the beginning of an event to its end.
Wall Time: Time taken for all invocations of method.
Percentage: Percentage of total time taken to execute all invocations of method.
CPU Time(ms): Time spent by CPU to execute method or transaction.
Thread Queue Time(ms): Thread time required.
Self-Time(ms): Time taken for the invocation of a method excluding the time taken by inner methods.
Thread Id: Unique Id of the thread.
Thread: Type of thread used.
Arguments: The path of the thread.
Instance: This denotes the instance running on that tier.
Class: Name of the class.
Captured Details: Show the details of the captured thread.

Note: Only instrumented methods are shown in the method call tree. There may be other non-instrumented methods called in between methods the have parent-child relationships in the method call tree.

You can download the method call details in Excel, Word, and PDF format.

Method call tree filter settings: Here, the user can configure the filter for Method Calling Tree. Upon clicking the Apply button, filters are applied, and Method Calling Tree is displayed accordingly. To apply the settings, user has to click on the Filters button. Once the user clicks on the filter button, the Method Call Tree Filter Settings window will occur as shown in the below Figure 49.

In the Method Call Tree Filter Settings window, the user has to fill the following fields which are mentioned below:

Show Methods (max depth):
Show Methods (walltime in ms): This option will show the methods walltime which will be greater than or equal to the provided value.
Highlight Methods (walltime in ms): This option will highlight the methods walltime which will be greater than or equal to the value provided.
Highlight Successor Methods (wrt elapsed time in ms): This option will highlight all the successor methods which will occur after the value specified by the user w.r.t elapsed time in milliseconds.

Positive Filters

In Positive Filters, the user has to choose value for the following fields:

Packages: In packages, the user has to select the java package from the drop-down list which they want to import for the method call.
Classes: In classes, the user has to select the required class as per the package which they want to import.
Methods: In classes, the user has to select the required class as per the package which they want to import.

Negative Filters

In Negative Filters, the user has to provide value on the below given field:

Do not show methods: In this option, the user has to provide the method name which they do not want to show.

Advanced Settings

In Advanced Settings, there are the following options from which the user can select as per their requirement:

Do not filter callouts: In the Do not filter callouts options, there are following options which the user can select which are listed below:
HTTP CallOuts
DB CallOuts
JMS CallOuts
Threads CallOuts
Exceptions
Async Callouts

Notes:

If a user wants to show the thread queue time and self-time, then the user has to select the Show Thread Queue Time and Self Time
If the user wants to expand the method in the positive filter option, then the user has to select the Expand Method Upto checkbox and has to provide the value as per their requirement. By default, the value is set to 25.
Once the user has filled up all the fields, the user has to click on the Apply button in order to apply the Method Call Tree Filter Settings. A user can also reset the Method Call Tree Filter settings by clicking on the Reset If the user does not want to apply the settings, then the user can click on the Cancel button.

Apart from these, the Method Call Details window displays various icons through which a user can perform various functions:

SR. No.	Icon	Description
1		Download Word Report
2		Download Excel Report
3		Download PDF Report
4		Show Callouts Expanded From
5		Dynamic Logging
6		Click to Start Auto Instrumentation
7		Click to get Flowpathinstance
8		Toggle Columns
9		Show Method Color Meaning

Hotspot

The hotspot details for a flowpath can be used to identify thread level performance bottlenecks in the form of hotspots. This feature is used to view the Hotspot thread details. This section displays information on

Thread Hotspot Stats
Hotspot Summary
Thread Stacktrace
Integration Point Calls

Thread Hotspot Stats

This displays two sections:

Hotspot thread by State: This displays hotspot thread based on various states, such as runnable, waiting, blocked, and timed waiting.
Hotspot thread by Stack depth: This displays hotspot thread based on stack depth.

HotSpot Summary

This section displays a summarized view of hotspots that includes:

Thread ID: Unique ID of the thread.
Thread Name: Name of the thread.
Tier: Shows the name of the tier.
Instance: This denotes the instance running on that tier.
Hotspot Entry Time: Start Time of the hotspot.
HotSpot Duration: Duration of the hotspot in seconds.
Total Hotspot Duration: Total duration of the hotspot.
Thread State: Current state of the thread, such as: runnable, timed waiting, blocked, running, etc.
Thread Priority: This denotes the priority which is being set by the user for the respective threads.
Stack Depth: This denotes the maximum size limit of the java stack for each java thread.

Thread Stacktrace

This section displays details of thread stack trace based on the selected thread ID. A user can also configure which column to display or hide by clicking the Show/Hide columns icon.

It has the following columns:

Method: It denotes the methods executed for that business transaction
Class: Shows the class of the particular thread.
Line: Shows the line number of the thread.
Source File: Enter the source of the file.
Elapsed Time: It is the amount of time that passes from the beginning of an event to its end.
Frame: Shows the number of frames.

Integration Point

This section displays details of integration point calls.

The following are the column in integration point:

Integration Point: Integration point is an end-point of the tier hierarchy. JDBC, Endeca Search, SOAP – Loyalty, PayPal, Bank Card, ONG Service Etc.
Discovered IP Name: It is the integration point name.
Type: Shows the IP individual info.
Total Duration(ms): Total time taken for IP calls.
Avg Duration(ms): Average time taken for IP calls.
Max Duration(ms): Maximum duration of IP calls.
Min Duration(ms): Minimum time taken in IP calls.
Total Count: Total number of IP calls.
Min Count: Minimum number of IP calls.
Max Count: Maximum number of IP calls.
Error(s): Errors count of IP calls.
Average Network Delay: Average time delay in IP calls.
Total Network Delay: Total delay in IP calls.

DB Queries

DB queries are used for extracting useful information from database in the form of result sets.

DB Stats

DB queries have following columns:

Query: Shows the query to be executed.
First Query Time: Time when the first query is entered.
Last Query Time: Time when the last query is entered.
Query Count: Number of query.
Error Count: Number of error count.
Min(ms): Minimum time taken for query.
Max(ms): Maximum time taken for query execution.

Query: Shows the name of the query and its description.

HTTP Report

HTTP reports have following options

HTTP Information
Request Headers
Request Body
Custom Data
Session Data
Condition Data

HTTP Information

Tier: This field shows the name of the tier.
Server: This field shows the name of the server.
Instance: This field shows the name of the instance.
Status Code: This field is used to show the status of the server.

Request Headers

Request header shows the header name and its corresponding value.

Request Body

Shows the configuration of the request body.

Response Header

Response header shows the configuration of the response header.

Response Body

Response body shows the response body configuration.

Custom Data

It shows the custom data and the configuration value.

Session Data

It shows the session data and the corresponding configuration value.

Condition Data

Shows the conditional data and its corresponding configuration value.

Exceptions

Exception instance report displays detailed information on the exception occurred. Exceptions have the following options:

Exception Stats
Aggregate Exception
Exceptions
Stack Trace

Exception Stats

This section displays two sub-sections. One for group by exceptions where percentage of exception groups (such as PSQLException and NumberFormatException) is displayed and another is exceptions where exception stats (exception by time at X-axis and exception by count at Y-axis) is displayed.

Aggregate Exceptions

This section provides aggregated details of the exception(s), such as exception class, throwing class, throwing method, and exception count.

Exception

This section displays details of an exception. To view details of a particular exception class, click the exception counts within the Aggregate Exception(s) section.

The details include are as follows:

Tier: A tier is a logical entity that constitutes a high-level grouping of individual nodes.
Server: This denotes the server name.
Instance: Denotes the instance running on that tier.
Integration point: Integration point is an end-point of the tier hierarchy.
Time: Time when the exception was created.
Exception Class: Shows the class of the exception.
Message: It shows the message of that exceptions.
Throwing class: Shows the class of the throwing exception.
Throwing method: Method of throwing exception.
Line No-Line in which the exception happens.
Cause– Shows the reason of the exception.

Stack Trace

In this exception stack trace, you see a list of chained exceptions which is sorted by the exception at the highest level to the one at the lowest level.

Method Timing

This report provides an intuitive view of the methods with their execution timings. This helps in getting the information, such as which methods are taking more time in the execution and causing an issue in the overall execution of the query or business transaction.

Method timing has the following option:

Methods Stats
Method Summary

Method Stats

This report displays the details of method timings in the form of Pie chart.

Method Summary

Package: It displays the package to which the called method belongs. A package is a group of similar types of classes, interfaces, and sub-packages.
Class: It displays the class to which the called method belongs. A class is a group of objects which have common properties.
Method: It is the method the user is analyzing. A method is a block of code, or a set of code grouped to perform a certain task or operation
Functional Group Method: Shows the name of the functional group method.
Fully Qualified Name with Signature: Shows qualified name with signature.
CumWallTime: Total time taken for all invocations of method.
Percentage: Percentage of the total time taken to execute all invocations of the method.
AvgWallTime: Average time taken for all invocations of method.
CumSelftime(sec): Total self-time taken for all invocations of method.
AvgSelfTime(ms): Average self-time taken for single invocation of method.
CumCpuSelfTime: Total CPU self-time taken for all invocations of method.
AvgCpuSelfTime(ms): Average CPU self-time taken for all invocations of method.
Count: Number of invocations of a method.
MinSelfTime(ms): Minimum self-time taken for single invocation of method.
MaxSelfTime(ms): Maximum self-time taken for single invocation of method.
WaitTime: Any thread consuming time on wait method.
Sync Time: Any thread stuck before synchronized block or synchronized method due to lock not being available.
IOTime(ms): Thread pause time due to input/output operations.
Suspension Time(ms): Thread pause time due to java GC.

IP Summary

This is an advanced BT Analyzer to capture all respective and individual Integration Point Calls. IP summary has the following options:

BT Callout Summary
BT Callout Details

BT Callout Summary

This is an advanced BT Analyzer to capture all respective and individual Integration Point Calls.

Business Transaction: This denotes the name of the business transaction executed.
Min HTTP Callout: Denotes the minimum number of calls in HTTP.
Max HTTP Callout: Maximum number of calls in HTTP.
Total HTTP Callout: Total number of calls in HTTP.
Min DB Callout: Shows the minimum number of calls in the database.
Max DB Callout: Maximum number of calls in the database.
Total DB Callout: Total number of calls in the database.
Min Callout: Minimum number of calls in this method.
Max Callout: Maximum number of calls in this method.
Flowpath: Flowpath consists of a controlled flow of fully qualified methods invocation.

BT Callout Details

Integration Point: Integration point is an end-point of the tier hierarchy. JDBC, Endeca Search, SOAP – Loyalty, PayPal, Bank Card, ONG Service Etc.
Discovered IP Name: It is the integration point name.
Type: Shows the IP individual info.
Total Duration(ms): Total time taken for IP calls.
Avg Duration(ms): Average time taken for IP calls.
Max Duration(ms): Maximum duration of IP calls.
Min Duration(ms): Minimum time taken in IP calls.
Total Count: Total number of IP calls.
Min Count: Minimum number of IP calls.
Max Count: Maximum number of IP calls.
Error(s): Errors count of IP calls.
Average Network Delay: Average time delay in IP calls.
Total Network Delay: Total delay in IP calls.

Transaction Flowmap

Once business transactions are refined, the user can monitor the business transaction performance. Business transaction list contains a high-level information of the application health and performance of a business transaction. It displays information such as, FP details, DB request report, method timing report, hotspot thread details, view related graphs, current instance log, and all instance log.

A user can view the transaction flow map that provides a detailed flow of the transactions. Network delay is also captured for each node of transactions and with respect to each method.

Sequence Diagram

A Sequence diagram is an interaction diagram that shows how objects operate with one another and in what order. It is a construct of a Message Sequence Chart. A sequence diagram shows object interactions arranged in time sequence. Sequence Diagram can be opened directly from flowpath using flowpath Instance or indirectly from Split View by first clicking on any link then on Sequence Diagram tab.

For filter settings, click the Filters button at the upper-right corner. This displays the Sequence Diagram Filter Settings window.

In the Sequence Diagram Filter Settings window, the user has to fill the following fields which are mentioned below:

Class Name Width (in px): In this option, the user has to provide the width for the class name. The value which the user will provide will be in Pixels (px).
Top Methods: In this option, the user has to provide the value.
Show Methods (walltime in ms): This option will show the methods walltime which will be greater than or equal to the provided value.
Highlight Methods (walltime in ms): This option will highlight the methods walltime which will be greater than or equal to the value provided.

Positive Filters

In Positive Filters, the user has to choose value for the following fields which are listed below:

Packages: In packages, the user has to select the java package from the drop-down list which they want to import for the method call.
Classes: In classes, the user has to select the required class as per the package which they want to import.
Methods: In classes, the user has to select the required class as per the package which they want to import.

Negative Filters

In Negative Filters, the user has to provide value on the below given field:

Do not show methods: In this option, the user has to provide the method name which they do not want to show.

Advanced Settings

In Advanced Settings, there are the following options from which the user can select as per their requirement:

Do not filter callouts: In the Do not filter callouts options, there are following options which the user can select which are listed below:
HTTP CallOuts
DB CallOuts
JMS CallOuts
Threads CallOuts
Exceptions
Async Callouts
Error CallOuts
DBError CallOuts

Notes:

If a user wants to show the thread queue time and self-time, then the user has to select the Show All
If the user wants to expand the method in the positive filter option, then the user has to select the Expand Method Upto checkbox and has to provide the value as per their requirement. By default, the value is set to 25.
Once the user has filled up all the fields, the user has to click on the Apply button to apply the Method Call Tree Filter Settings. A user can also reset the Method Call Tree Filter settings by clicking on the Reset If the user does not want to apply the settings, then the user can click on the Cancel button.

Apart from these, the Method Call Details window displays various icons through which a user can perform various functions:

SR. No.	Icon	Description
1		Click to Start Auto Instrumentation/ Dynamic Diagnostics
2.		View Image in New Tab

Dynamic Diagnostics Configuration

The auto instrumentation process is time-consuming, as it involves taking in-memory thread dumps at regular intervals and identifying all methods in active threads. To address this limitations, Dynamic diagnostics auto instrumentation is implemented. It is a lightweight solution to finding problems in business transactions. Dynamic Diagnostics instrumentation is triggered solely for specific business transaction, and stack dumps are considered only for those threads, which are executing those transaction.

There are two ways of configuring Dynamic Diagnostics:

General Settings
Advance Settings

General Settings

It has the following fields:

Business Transaction: Name the business transaction to be used.
Session name: Name of the session, which is automatically named, based on the business transaction selected. User can edit the business transaction.
Session Duration: Specify the duration of the session for running.
Completion Mode: This refers to the is the action that need to be, taken regarding downloading the instrumentation files, once the instrumentation process is completed. There are following options:
No Download: No files are downloaded.
Download files after session: Files are downloaded after session is completed.
Download and apply: Files are downloaded and applied to the instance.
Apply Mode: There are following options within this section:
Apply on Instance: Instrumentation is applied on instances without saving the file.
Apply on Instance and Save: Instrumentation is applied on instance and the file is saved.

Advance Settings

We have the following fields in advanced settings:

Maximum stack depth to identify idle threads: In Auto Instrumentation, application agents will ignore idle threads, which have stack depth less than the provided value.
Sample Interval: Shows the sample interval of dynamic diagnostics.
Trace Level: It is the trace level of DD auto instrumentation.
Number of thread samples to instrument methods: In every cycle of provided thread sample count, important method(s) will be instrumented.
Percentage of executing methods in thread samples to be instrumented: The percentage of methods that will be instrumented from the list of all considered methods.
Percentage of executing methods in flow paths to be un–instrumented: The percentage of the methods that will be un-instrumented if their response time is very small.
Maximum method(s) to be considered for Auto Instrumentation: Maximum unique method(s) without overloading which will be considered in Auto Instrumentation process.
Minimum average execution time of method for Instrumentation (in ms): Methods will be un-instrumented if the average execution time of the method is less than the provided value.
Weightage of methods in stack traces having Exception/Integration point calls: All methods in stack traces having exception and integration point calls will have more weightage for Auto Instrumentation.
Concurrent Auto Instrumentation Threshold: Maximum number of active business transaction serving threads for DD AI session.
Dump data: Shows the quantity of data dump.
Thread blacklist filter: Provide the thread names excluded from the DD AI session.
Thread whitelist filter: Provide the thread names included in the DD AI session.
Blacklist filter: Provide BT names excluded from DD AI session.
Whitelist filter: Provide BT names included in the DD AI sessions.
Remove Auto Instrumentation session changes after its completion: Un-instrument all methods, which are considered for Auto Instrumentation only after Auto Instrumentation session completion.
Delete from server: Deletion of file from the server.
Check the box to enable Auto instrumentation for method capturing.

Flowpaths by Callout Errors

This feature is used to view the Flowpaths by callout errors. This displays the top flowpath with category as an error. To do this, follow the below-mentioned steps:

Right-click on a tier, go to DrillDown, and click Flowpaths by Callout Errors

2. The Flowpath report is displayed. The report displays the top flowpaths with callout.

It has the following fields:

Tier: This denotes the name of the tier.
Server: This denotes the server’s name
Instance: This denotes the instance running on that tier.
Business Transaction: Denotes the name of the business transaction to be executed.
FlowpathInstance: It is the flowpathinstance number. On clicking this, user can see the sequence diagram page.
URL: It denotes the page URL of the business transaction.
Category: It denotes the category of business transaction, such as normal, slow, very slow.
Start Time: This denotes the start time of the business transaction or method.
Total Response Time (ms): This denotes the time taken in the execution of the business transaction.
CPU Time: Time spent by the CPU to execute method or transaction.
Methods: It denotes the number of methods executed for that business transaction.
Callouts: It denotes the number of calls for that method.
Callout Errors: It denotes the number of errors that occurred in the method invocation.
DB Callouts: It denotes the number of calls for that database.
Status: This denotes the status/response code of the business transaction, such as 200, 500.
Corr ID: This denotes the correlation ID.
ND Session ID: Denotes ND session ID.
NV Page ID: Shows NV Page ID, if any.
Coherence Callout: Number of calls in coherence.
JMS Callout: Number of calls in JMS.
NV Session ID: Shows the NV session ID.
Wait Time(ms): Any thread consuming time on wait method.
Sync Time(ms): Any thread stuck before the synchronized block or synchronized method due to the lock not being available.
IO Time(ms): Thread pause time due to input/output operations
Suspension Time(ms): Thread pause time due to java GC.
Store ID: Denotes store ID.
Terminal ID: Shows the terminal ID.
QTime(ms): Query time in ms.
Response Time(ms): This denotes the time taken in the execution of business transaction.
Parent FlowpathInstance: Parent flowpath instance number
Total CPU Time(ms): Total CPU time taken for all invocations of the method.
GC Pause: It is the Garbage collection pause.
Flag: This displays the detailed information of the applied overlay.
Application Name: Flowpath application name.